<html>
<head><meta charset="utf-8"><title>Scaling Analyzer to Google-sized codebases · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html">Scaling Analyzer to Google-sized codebases</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="158875724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158875724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158875724">(Feb 19 2019 at 12:02)</a>:</h4>
<p>I've recently watched the talk about clangd (<a href="https://www.youtube.com/watch?v=5HIyAXj1YNQ" target="_blank" title="https://www.youtube.com/watch?v=5HIyAXj1YNQ">https://www.youtube.com/watch?v=5HIyAXj1YNQ</a>), and it's significantly different from other IDE talks in that it places emphasis on working with behemoth codebases. I don't think we need to solve  this problem right now (getting precise completions for a single file with traits is much more important), but it's something to keep in mind.</p>
<p>The trick they use is to split IDE into "analyze the current translation unit" and "analyzer the whole workspace" bits. Analyze TU "just" runs clang-frontend to get completions for methods, etc. Analyze workspace uses offline, potentially remote indices. </p>
<p>In Rust, I fear we might get into trouble with just the first bit. In C++, because of header files, it should be pretty easy to analyzer only "interface" of the dependencies, skipping cpp files altogether. In Rust, we don't have header files, so kinda need to analyzer the whole transitive closure of code. </p>
<p>It might be cool to introduce a notion of "crate interface file" (not expressible in the surface syntax) so that we can break dependency chain. I imagine a reified "public interface of the crate" will be useful for public-private dependencies as well .</p>
<div class="youtube-video message_inline_image"><a data-id="5HIyAXj1YNQ" href="https://www.youtube.com/watch?v=5HIyAXj1YNQ" target="_blank" title="https://www.youtube.com/watch?v=5HIyAXj1YNQ"><img src="https://i.ytimg.com/vi/5HIyAXj1YNQ/default.jpg"></a></div>



<a name="158886461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158886461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158886461">(Feb 19 2019 at 14:43)</a>:</h4>
<p>Yeah, I think saving &amp; loading the item map etc. would actually make sense, already to avoid parsing all of std on every startup. The compiled crates from rustc are something like that, aren't they? Though we'd need a more stable format, of course.</p>
<p>And I feel like any big project will be split into many small crates, so this might already be enough.</p>



<a name="158887371"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158887371" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158887371">(Feb 19 2019 at 14:52)</a>:</h4>
<p>The problem with item-map is that it contains all of the private items as well.  And splitting into crates does not fundamentally change the picture: you still have to process the transitive closure of reachable code. </p>
<p>In (non-templated) C++ headers the situation is different: you can use tricks like pimpl to make sure that you don't even have to look at the implementation code. Moreover, the impl can change freely without affecting the header.</p>
<p>I guess we can do similar with the item-map setup, if we add a <code>public API</code> projection query.</p>



<a name="158889034"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158889034" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158889034">(Feb 19 2019 at 15:11)</a>:</h4>
<p>we would need the private items to provide a "item is private" diagnostic instead of "item not found" though, especially if we want to have an assist that turns it public for local crates, right? <span aria-label="thinking" class="emoji emoji-1f914" role="img" title="thinking">:thinking:</span></p>



<a name="158889484"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158889484" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158889484">(Feb 19 2019 at 15:16)</a>:</h4>
<p>Good point! I think it woudn't be too bad to say "item not found" for completely external crates, but doing things <em>differently</em> for local and remote crates seems hard.</p>
<p>And, given that item-map lacks item bodies, perhaps I am trying to solve a non-existing problem? Like, if you want to <strong>link</strong> the resulting library/binary, than you need to be able to process everything (including bodies) on a single machine.</p>



<a name="158889990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158889990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158889990">(Feb 19 2019 at 15:24)</a>:</h4>
<p>do we currently have statistics about the memory usage of the item maps? I don't think so, right?</p>



<a name="158890083"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158890083" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158890083">(Feb 19 2019 at 15:25)</a>:</h4>
<p>Yeah, we don't have it, but shouldn't be too hard to add</p>



<a name="158890182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158890182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158890182">(Feb 19 2019 at 15:27)</a>:</h4>
<p><a href="https://github.com/Aeledfyr/deepsize" target="_blank" title="https://github.com/Aeledfyr/deepsize">https://github.com/Aeledfyr/deepsize</a> or heap_size_of could be useful</p>



<a name="158903959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158903959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158903959">(Feb 19 2019 at 18:15)</a>:</h4>
<p>I feel like (a) this is often the role that nrc was citing for save analysis but (b) it seems like this would be achieved by having a "outer frontier" of query results that we can preserve from crates and things (without having to preserve all the detailed work needed to reproduce that frontier).  In other words, I'd really like to express this idea in terms of <strong>queries</strong></p>



<a name="158963965"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158963965" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Dale Wijnand <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158963965">(Feb 20 2019 at 10:29)</a>:</h4>
<blockquote>
<p>In Rust, I fear we might get into trouble with just the first bit. In C++, because of header files, it should be pretty easy to analyzer only "interface" of the dependencies, skipping cpp files altogether. In Rust, we don't have header files, so kinda need to analyzer the whole transitive closure of code. </p>
<p>It might be cool to introduce a notion of "crate interface file" (not expressible in the surface syntax) so that we can break dependency chain. I imagine a reified "public interface of the crate" will be useful for public-private dependencies as well .</p>
</blockquote>
<p>Another option is doing a two-phase compilation, first outlining the API and later compiling the bodies.  See this presentation by Twitter where they're exploring this in an experimental compiler for Scala: <a href="https://www.youtube.com/watch?v=8SnIBkJXD8I" target="_blank" title="https://www.youtube.com/watch?v=8SnIBkJXD8I">https://www.youtube.com/watch?v=8SnIBkJXD8I</a>.</p>
<div class="youtube-video message_inline_image"><a data-id="8SnIBkJXD8I" href="https://www.youtube.com/watch?v=8SnIBkJXD8I" target="_blank" title="https://www.youtube.com/watch?v=8SnIBkJXD8I"><img src="https://i.ytimg.com/vi/8SnIBkJXD8I/default.jpg"></a></div>



<a name="158967877"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158967877" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158967877">(Feb 20 2019 at 11:49)</a>:</h4>
<p>This two-phase setup basically falls out from the query system, at least if we set things up correctly -- but it's a good thing for us to pay attention to!</p>



<a name="158970599"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Scaling%20Analyzer%20to%20Google-sized%20codebases/near/158970599" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Dale Wijnand <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Scaling.20Analyzer.20to.20Google-sized.20codebases.html#158970599">(Feb 20 2019 at 12:38)</a>:</h4>
<p>OK, cool! <span aria-label="smile" class="emoji emoji-263a" role="img" title="smile">:smile:</span></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>